{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 20 - Correlation, causation, and heat maps\n",
    "\n",
    "The Federal Reserve Bank of New York has information about the labor market for recent college graduates [here](https://www.newyorkfed.org/research/college-labor-market/college-labor-market_compare-majors.html).\n",
    "\n",
    "The data in this table can be downloaded as an Excel file at the bottom of the page.  If you open this file in Excel, you can save the last table as a CSV file.  Alternatively, download the data as a CSV file from the course website. \n",
    "\n",
    "We will be using a new data visualization library called [Seaborn](https://seaborn.pydata.org/#).  Install it with the code below (this might take a few minutes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!pip install --user seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import Seaborn and the other libraries so we can use them in our code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading and cleaning the data\n",
    "\n",
    "Before creating the dataframe, look at the CSV file.  Do you notice anything that you will have to account for when reading in the data?  \n",
    "\n",
    "Hint:  To skip 5 rows at the end of the CSV file, use the parameter `skipfooter = 5`.  (Ignore the warning.)\n",
    "\n",
    "Load the labor market CSV file into a dataframe called `labor`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display your `labor` dataframe below to check it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will want to use all the columns as numbers, but has `read_csv` interpreted them as numbers?  Type `labor.dtypes` below and run the code to see what *type* each column is.  `dtypes` is a property of the dataframe, not a function, so it doesn't have `()` at the end.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decimal numbers are stored as `float64`, so the `Unemployment Rate`, `Underemployment Rate`, and `Share with Graduate Degree` have been read in as numbers.  The `Median Wage Early Career` and `Median Wage Mid-Career` columns have type `object` which probably means a string.  These columns were read in as strings because they contains commas.\n",
    "\n",
    "The following code will remove the commas and tell Pandas that these columns are `float`s as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "labor[\"Median Wage Early Career\"] = labor[\"Median Wage Early Career\"].str.replace(\",\",\"\").astype(float)\n",
    "labor[\"Median Wage Mid-Career\"] = labor[\"Median Wage Mid-Career\"].str.replace(\",\",\"\").astype(float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that this code worked by displaying `labor` again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the type for each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlation matrix\n",
    "\n",
    "We can compute the correlation matrix with the code `labor.corr()`.  Try it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which pair of columns are the most correlated?  Which pair of columns are the least correlated?\n",
    "\n",
    "### Heat maps\n",
    "\n",
    "We can visualize the correlation matrix with a heat map.  To do this, we need to save the correlation matrix in a variable (say `corr_matrix`) and then run the code `sns.heatmap(corr_matrix)`.  Try it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the pairs of columns that you thought were most correlated and least correlated above.  Do the colors for these pairs make sense?\n",
    "\n",
    "Make a scatter plot of the two columns that are most correlated.  Remember the code pattern for creating a scatter plot is `df.plot.scatter(x = \"column name 1\", y = \"column name 2\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does the relationship look linear (like a line)?\n",
    "\n",
    "Now make a scatter plot of the two columns that were least correlated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does this relationship look linear?\n",
    "\n",
    "We can plot all possible scatter plots at once with the Seaborn command `sns.pairplot(labor)`.  Try it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you spot the pairs of columns with negative correlation from the plots?  What plots are on the diagonal?\n",
    "\n",
    "## Another heat map: Traffic counts\n",
    "\n",
    "To fully see the power of heat maps, we need a larger data set.  Download the dataset of traffic counts [here](https://data.cityofnewyork.us/Transportation/Traffic-Volume-Counts-2012-2013-/p424-amsu).\n",
    "\n",
    "This dataset contains counts of the number of vehicles to pass different sections of road at different times of the day.\n",
    "\n",
    "Load the CSV file into a variable called `traffic`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display your new dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the columns?\n",
    "\n",
    "Compute the correlation matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our correlation matrix is fairly large, so it is hard to see patterns in it.  To visualize the correlation matrix, display it as a heatmap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the heatmap?  Which columns are similar?  Does this make sense?\n",
    "\n",
    "### Challenges:\n",
    "- Which pair of numeric columns in the taxi dataset are the most correlated?  Which pair of numeric columns in the taxi dataset are the least correlated?  Do these results make sense?\n",
    "- Which pair of numeric columns in the Starbucks drink nuitrition information are the most correlated?  Which pair are the least correlated?  Do these results make sense?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}